Skip to content

perf: improve wildcard query perf with predicate and contains-check pushdown #397

Merged
cheb0 merged 8 commits into
mainfrom
0-wildcard-predicate-pushdown
Jun 5, 2026
Merged

perf: improve wildcard query perf with predicate and contains-check pushdown #397
cheb0 merged 8 commits into
mainfrom
0-wildcard-predicate-pushdown

Conversation

@cheb0

@cheb0 cheb0 commented Apr 3, 2026

Copy link
Copy Markdown
Collaborator

Description

Currently we spend only a fraction of time calling bytes.Index. This PR partially addresses that.

This PR pushes pattern.Searcher to Block level, so that Block is able to stream tokens through searcher. For ordinary wildcards like *error* there is direct FindContains method which is even faster.

For example, query message:*foobarf*:
main: 86 ms
using FindToken: 50 ms
using FindContains: 37 ms

So, FindContains just throws out costly abstractions to get additional performance. We could also provide a dedicated func like FindSuffix, for example. This is a typical example when performance requires additional code.

Query Type Ids cold, ms hot, ms cold (branch), ms hot (branch), ms cold diff hot diff
trace_id:*foobar reg 0 18.76 4.37 16.14 1.84 -14% -57.9%
k8s_pod:*6 reg 100 13.3 0.67 13.03 0.47 -2% -29.9%
message:*err* reg 100 138.72 26.97 120.27 12.36 -13.3% -54.2%
message:*foo* reg 100 77.69 27.08 60.54 11.84 -22.1% -56.3%
message:*request* reg 100 124.95 25.45 104.13 10.37 -16.7% -59.3%
message:*foobar*foobar* reg 0 187.54 64.25 147.31 30.5 -21.5% -52.5%
message:*foobarfoobar* reg 0 184.93 63.87 121.51 20.39 -34.3% -68.1%
message:*very_very_message_aggregator_events* reg 0 173.45 51.62 116.9 12.81 -32.6% -75.2%

Next steps:

  • try calling bytes.Index over Block payload - already shows good results
  • build Offsets lazy - if previous is done
  • modernize token Block, boost Unpack speed
  • I have read and followed all requirements in CONTRIBUTING.md;
  • I used LLM/AI assistance to make this pull request;

@codecov-commenter

codecov-commenter commented Apr 3, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.50575% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.70%. Comparing base (0a1f2c1) to head (c61bdab).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
frac/active_token_list.go 78.94% 2 Missing and 2 partials ⚠️
frac/sealed/token/block_loader.go 86.66% 1 Missing and 1 partial ⚠️
frac/sealed/token/provider.go 90.00% 1 Missing and 1 partial ⚠️
pattern/pattern.go 91.30% 1 Missing and 1 partial ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #397      +/-   ##
==========================================
+ Coverage   70.58%   70.70%   +0.12%     
==========================================
  Files         222      222              
  Lines       17375    17441      +66     
==========================================
+ Hits        12264    12332      +68     
+ Misses       4195     4191       -4     
- Partials      916      918       +2

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

dkharms
dkharms reviewed Apr 3, 2026
View reviewed changes
Comment thread frac/sealed/token/block_loader.go Outdated
dkharms
dkharms reviewed Apr 3, 2026
View reviewed changes
Comment thread frac/sealed/token/block_loader.go Outdated
@eguguchkin eguguchkin self-requested a review April 6, 2026 10:20
@eguguchkin eguguchkin modified the milestones: v0.72.0, v0.73.0 Apr 13, 2026
@cheb0 cheb0 added the performance Features or improvements that positively affect seq-db performance label May 12, 2026
@eguguchkin eguguchkin modified the milestones: v0.73.0, v0.72.0 May 18, 2026
dkharms
dkharms reviewed May 20, 2026
View reviewed changes
Comment thread frac/sealed/token/block_loader.go Outdated
Comment thread pattern/pattern.go Outdated
Comment thread frac/sealed/token/provider.go Outdated
Comment thread frac/sealed/token/provider.go Outdated
Comment thread frac/sealed/token/provider.go Outdated
eguguchkin
eguguchkin approved these changes May 21, 2026
View reviewed changes
Comment thread pattern/pattern.go
Comment thread frac/sealed/token/provider.go Outdated
Comment thread frac/sealed/token/provider.go Outdated
@cheb0 cheb0 merged commit 985c744 into main Jun 5, 2026
8 checks passed
@cheb0 cheb0 deleted the 0-wildcard-predicate-pushdown branch June 5, 2026 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dkharms dkharms dkharms approved these changes

@eguguchkin eguguchkin eguguchkin approved these changes

Assignees

No one assigned

Labels

performance Features or improvements that positively affect seq-db performance

Projects

None yet

Milestone

v0.72.0

Development

Successfully merging this pull request may close these issues.

4 participants

@cheb0 @codecov-commenter @dkharms @eguguchkin